1. Technical Field
The present invention generally relates to augmented reality and, in particular, to a method for augmented reality guided instrument positioning with depth determining graphics.
2. Background Description
Augmented reality, also commonly referred to as augmented vision or augmented reality vision, augments an observer's view of the real world by superimposing computer generated graphical information. This information may be as simple as a text label attached to some object in the scene, or as complex as a 3D model of a patient's brain derived from an MRI scan and aligned to the real view of the person's head.
The observer may observe a real scene directly with his or her eyes, with the additional graphical information being blended therewith via a semi-transparent display located between the observer and the real scene. Such a display device can be, for example, a see-through head mounted display.
The display can also be opaque, like a computer screen or a non-see-through head mounted display. Such a display then presents to the observer the complete augmented view, i.e., a combination of the real-world view and the graphics overlay. A video camera takes the place of the real-world observer to capture the real world-view. For stereo vision, two cameras are required. A computer is used to combine the live video with the graphics augmentation.
The graphics have to be positioned, oriented, and scaled, or even rendered in a perspective fashion for correct alignment with the real-world view. It is desirable to “anchor” the graphics to a real-world object. To do this, the position and orientation of the camera with respect to the object, as well as the orientation of the object, must be known. That is, the relationship between two coordinate systems, one corresponding to the camera and the other corresponding to the object, must be known.
Tracking denotes the process of keeping track of the preceding relationship. Commercial tracking systems are available that are based on optical, mechanical, magnetic, inertial, and ultrasound measurement principles.
FIG. 1 is a block diagram illustrating an augmented reality system 100 wherein video images of the real world are combined with computer generated graphics, according to the prior art. The system 100 includes: a video camera 110; external trackers 112; 2D/3D graphics module 114; an image processing module 116; a pose calculation module 118; a graphics rendering module 120; a video and graphics overlay module 122; and a display 124. As is known, a 3D perception may be achieved through the use of two cameras and a stereo display.
The operation of the elements typically employed in an augmented reality system as well as the calibration required of such a system is described by: Ahlers et al., in “Calibration Requirements and Procedures for a Monitor-based Augmented Reality System”, IEEE Transactions on Visualization and Computer Graphics, 1(3): 255-273, 1995; Navab et al., in “Single Point Active Alignment Method (SPAAM) for Calibrating an Optical See-through Head Mounted Display”, Proc. of the IEEE International Symposium on Augmented Reality, ISAR '00, Munich, Germany, October 2000; and Sauer et al., “Augmented Workspace: Designing an AR Testbed”, Proc. of the IEEE International Symposium on Augmented Reality, ISAR '00, Munich, Germany, October 2000.
Augmented reality visualization can guide a user in manual mechanical tasks. For machine repair and maintenance scenarios, it has been suggested to augment the view with graphical pointers that show, e.g., which button to press or which screw to turn. Augmented reality visualization is also being suggested for medical applications where, e.g., biopsy needles have to be inserted into a target tumor without harming nearby nerves or where screws have to be inserted into bones at a precise location and in a precise direction.
A description of one of the main problems with augmented reality visualization, i.e., occlusion, will now be given. As noted above, augmented Reality visualization places virtual objects (computer generated graphics) into real scenes. The tracking of the vantage point, from which the real scene is viewed, with respect to a world coordinate system anchored at real world objects, allows the virtual objects to appear at desired locations in this world coordinate system. However, a correct visual interaction between real and virtual objects generally requires 3D information about the real objects. Disadvantageously, this 3D information is usually not available and, thus, the virtual objects are simply superimposed onto the image of the real scene. Accordingly, real objects can be hidden by virtual objects, although virtual objects cannot be hidden by real objects.
To illustrate the problem of occlusion, consider the following example: a simple virtual object such as a cube is superimposed onto a real scene, with all the real objects being farther away from the observer than the virtual cube. If this augmented scene is shown to the observer in stereo, the observer can easily perceive the correct 3D position of the cube. Now the observer inserts his hand in the scene so that his hand is in the foreground and occludes the position where the virtual cube is supposed to be. Since the system has no knowledge of the hand, the system still shows the virtual cube superimposed onto the scene image. The inability of the (real) hand to occlude the virtual cube confuses the 3D perception of the observer. The observer receives conflicting depth cues: the stereo depth cues tell the observer that the cube is behind the hand, the occlusion cue tells the observer that the cube must be in front of the hand (because otherwise it would be hidden by the hand).
Accordingly, it would be desirable and highly advantageous to have a method for augmented reality guided instrument positioning which is not adversely affected by occlusion.